The loss surfaces of neural networks with general activation functions

The loss surfaces of deep neural networks have been the subject several studies, theoretical and experimental, over last few years. One strand work considers complexity, in sense local optima, high dimensional random functions with aim informing how optimisation methods may perform such complicated settings. Prior Choromanska et al (2015) established a direct link between training multi-layer perceptron spherical multi-spin glass models under some very strong assumptions on network its data. In this work, we test validity approach by removing undesirable restriction to ReLU activation functions. doing so, chart new path through spin complexity calculations using supersymmetric Random Matrix Theory which prove useful other contexts. Our results shed light both strengths weaknesses context.